System and method for catheter detection in fluoroscopic images and updating displayed position of catheter

ABSTRACT

A method and system implementing a method for detecting a catheter in fluoroscopic data and updating a displayed electromagnetic position of the catheter on a 3D rendering is provided including navigating a catheter to a target area and acquiring fluoroscopic data from a fluoroscopic sweep of the target area. An initial catheter detection is performed to detect catheter tip candidates in each 2D frame of the fluoroscopic data using a shallow neural network. A secondary catheter detection is performed to detect catheter tip candidates in each 2D frame of the fluoroscopic data using a deep neural network. False-positive catheter tip candidates are removed by reconstructing a 3D position of the catheter tip and finding an intersecting point of rays corresponding to each 2D frame.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of the filing date of provisionalU.S. Patent Application No. 62/627,911, filed Feb. 8, 2018, the entirecontents of which are incorporated herein by reference.

BACKGROUND

The disclosure relates to a system, apparatus, and method of detecting acatheter in a series of fluoroscopic images, determining the position ofthe catheter based on the detection, and updating the displayed positionof the catheter in an electromagnetic navigation system for surgicalprocedures.

There are several commonly applied methods for treating various maladiesaffecting organs including the liver, brain, heart, lung and kidney.Often, one or more imaging modalities, such as magnetic resonanceimaging, ultrasound imaging, computed tomography (CT), as well as othersare employed by clinicians to identify areas of interest within apatient and ultimately targets for treatment.

An endoscopic approach has proven useful in navigating to areas ofinterest within a patient, and particularly so for areas within luminalnetworks of the body such as the lungs. To enable the endoscopic, andmore particularly the bronchoscopic, approach in the lungs,endobronchial navigation systems have been developed that use previouslyacquired MRI data or CT image data to generate a three dimensionalrendering or volume of the particular body part such as the lungs. Inparticular, previously acquired images, acquired from an MRI scan or CTscan of the patient, are utilized to generate a three dimensional orvolumetric rendering of the patient.

The resulting volume generated from the MRI scan or CT scan is thenutilized to create a navigation plan to facilitate the advancement of anavigation catheter (or other suitable device) through a bronchoscopeand a branch of the bronchus of a patient to an area of interest.Electromagnetic tracking may be utilized in conjunction with the CT datato facilitate guidance of the navigation catheter through the branch ofthe bronchus to the area of interest. In certain instances, thenavigation catheter may be positioned within one of the airways of thebranched luminal networks adjacent to, or within, the area of interestto provide access for one or more medical instruments.

A fluoroscopic imaging device is commonly located in the operating roomduring navigation procedures. The standard fluoroscopic imaging devicemay be used by a clinician to visualize and confirm the placement of atool after it has been navigated to a desired location. However,although standard fluoroscopic images display highly dense objects suchas metal tools and bones as well as large soft-tissue objects such asthe heart, the fluoroscopic images have difficulty resolving smallsoft-tissue objects of interest such as lesions or tumors. Further, thefluoroscope image is only a two dimensional projection. In order to beable to see small soft-tissue objects in three dimensional space, anX-ray volumetric reconstruction is needed. Several solutions exist thatprovide three dimensional volume reconstruction of soft-tissues such asCT and Cone-beam CT which are extensively used in the medical world.These machines algorithmically combine multiple X-ray projections fromknown, calibrated X-ray source positions into three dimensional volumein which the soft-tissues are visible.

SUMMARY

The disclosure relates to a system, apparatus, and method of detecting acatheter in a series of fluoroscopic images, determining the position ofthe catheter based on the detection, and updating the displayed positionof the catheter in an electromagnetic navigation system for surgicalprocedures. The disclosure utilizes a combination of shallow neuralnetwork operators and deep neural network operators to detect cathetercandidates in a fluoroscopic data set. Additionally, false-positivecandidate detections are eliminated according to the methods describedherein. The position data of the catheter is acquired from thefluoroscopic data and is used as a correction factor for the displayedelectromagnetically tracked position of the catheter.

The system of the disclosure constructs a fluoroscopic-based 3Dconstruction of a target area which includes the catheter and a target(e.g., soft-tissue object, lesion, tumor, etc.) in order to determinethe location of the catheter relative to the target. In particular, thesystem identifies the position, orientation, angle, and distance of thecatheter relative to the target in each fluoroscopic frame of thefluoroscopic data. This relative location data is used to update adisplayed electromagnetic position of the catheter over a CT-basedrendering, for example, of a patient's luminal network. With thisupdated display, a clinician is able to more accurately navigate andconfirm placement of the catheter and other surgical tools relative to atarget during an electromagnetic navigation procedure.

Aspects of the disclosure are described in detail with reference to thefigures wherein like reference numerals identify similar or identicalelements. As used herein, the term “distal” refers to the portion thatis being described which is further from a user, while the term“proximal” refers to the portion that is being described which is closerto a user.

According to one aspect of the disclosure, a method for detecting acatheter in fluoroscopic data is provided. The method includes acquiringfluoroscopic data from a fluoroscopic sweep of a target area. The targetarea may be, for example, within a patient's luminal network. Thefluoroscopic data includes 2D fluoroscopic frames of the target areacaptured from different perspectives. The method further includesperforming an initial catheter detection for catheter tip candidates ineach 2D frame of the fluoroscopic data, performing a secondary catheterdetection for catheter tip candidates in each 2D frame of thefluoroscopic data. Additionally, the method includes eliminatingfalse-positive catheter tip candidates of the secondary catheterdetection by reconstructing a 3D position of the catheter tip andfinding an intersecting point of rays corresponding to each 2D frame,and reweighing the catheter tip candidates of the secondary catheterdetection based on a distance of the catheter tip candidate from aprojected 3D point.

The initial catheter detection may include applying a shallow neuralnetwork operator and the secondary catheter detection may includeapplying a deep neural network operator. The secondary catheterdetection for catheter tip candidates in each 2D frame of thefluoroscopic data may include considering the catheter tip candidates ofthe initial catheter detection. Reweighing the catheter tip candidatesof the secondary catheter detection based on a distance of the cathetertip candidate from a projected 3D point may include decreasing a weightof a pixel corresponding to a candidate when the distance of thecatheter tip candidate is far from the projected 3D point.

Additionally, the method may further include iteratively repeatingeliminating false-positive detections by reconstructing a 3D position ofthe catheter tip and finding an intersecting point of rays correspondingto each 2D frame. Additionally, or alternatively, the method may includedisplaying a user interface for manually selecting the catheter tip in a2D fluoroscopic frame of the fluoroscopic data prior to performing aninitial catheter detection for catheter tip candidates in each 2D frameof the fluoroscopic data.

In another aspect, a method for detecting a catheter in fluoroscopicdata during a surgical navigation procedure is provided. The methodincludes tracking an electromagnetic position of a catheter usingelectromagnetic coordinates during a navigation procedure of thecatheter to a target area, displaying the tracked electromagneticposition of the catheter on a display of a 3D rendering, and acquiringfluoroscopic data from a fluoroscopic sweep of the target area. Thefluoroscopic data includes 2D fluoroscopic frames of the target areacaptured from different perspectives. The target area may be, forexample, within a patient's luminal network and the 3D rendering may be,for example, a 3D rendering of the patient's luminal network. The methodfurther includes constructing fluoroscopic-based three dimensionalvolumetric data of the target area from the acquired fluoroscopic dataincluding a three-dimensional construction of a soft-tissue target inthe target area, acquiring, for each 2D frame of the fluoroscopic data,data of a position of the catheter relative to the three-dimensionalconstruction of the soft-tissue, and registering the acquired data ofthe position of the catheter relative to the three-dimensionalconstruction of the soft-tissue with the electromagnetic position of thecatheter. Additionally, the method includes displaying the position ofthe catheter on the display of the 3D rendering based on theregistration of the acquired data of the position of the catheterrelative to the three-dimensional construction of the soft-tissue withthe electromagnetic position of the catheter.

The method may further include performing at least one catheterdetection for catheter tip candidates in each 2D frame of thefluoroscopic data. An initial catheter detection may include applying ashallow neural network operator and a secondary catheter detection mayinclude applying a deep neural network operator. Additionally, oralternatively, the method may further include eliminating false-positivecatheter tip candidates of the secondary catheter detection byreconstructing a 3D position of the catheter tip and finding anintersecting point of rays corresponding to each 2D frame. The secondarycatheter tip detection may consider the candidates identified in theinitial catheter detection. Additionally, or alternatively, the methodmay include reweighing the catheter tip candidates of the secondarycatheter detection based on a distance of the catheter tip candidatefrom a projected 3D point.

In another aspect, a system for performing an electromagnetic surgicalnavigation procedure is provided. The system includes an electromagnetictracking system having electromagnetic tracking coordinates, a catheterincluding a sensor configured to couple to the electromagnetic trackingsystem for detecting a position of the catheter in the electromagneticcoordinates, and a computing device operably coupled to theelectromagnetic tracking system and the catheter.

The computing device is configured to display a navigation path to guidenavigation of the catheter to a target area, display the position of thecatheter in the electromagnetic coordinates on a 3D rendering, andacquire fluoroscopic data from a fluoroscopic sweep of the target area.The target area may be, for example, within a patient's luminal networkand the 3D rendering may be, for example, a 3D rendering of thepatient's luminal network. The fluoroscopic data includes 2Dfluoroscopic frames of the target area captured from differentperspectives. Additionally, the computing device is configured toperform an initial catheter detection for catheter tip candidates ineach 2D frame of the fluoroscopic data, perform a secondary catheterdetection for catheter tip candidates in each 2D frame of thefluoroscopic data, eliminate false-positive catheter tip candidates ofthe secondary catheter detection by reconstructing a 3D position of thecatheter tip and finding an intersecting point of rays corresponding toeach 2D frame, and reweigh the catheter tip candidates of the secondarycatheter detection based on a distance of the catheter tip candidatefrom a projected 3D point.

The computing device may be configured to perform an initial catheterdetection for catheter tip candidates in each 2D frame of thefluoroscopic data by applying a shallow neural network operator andperform a secondary catheter detection for catheter tip candidates ineach 2D frame of the fluoroscopic data by applying a deep neural networkoperator. Additionally, or alternatively, the computing device may beconfigured to construct fluoroscopic-based three dimensional volumetricdata of the target area from the acquired fluoroscopic data. Thefluoroscopic-based three dimensional volumetric data includes athree-dimensional construction of a soft-tissue target in the targetarea.

In an aspect, the computing device is further configured to acquire, foreach 2D frame of the fluoroscopic data, data of a position of thecatheter relative to the three-dimensional construction of thesoft-tissue, register the acquired data of the position of the catheterrelative to the three-dimensional construction of the soft-tissue withthe electromagnetic position of the catheter, and display the positionof the catheter on the display of the 3D rendering based on theregistration of the acquired data of the position of the catheterrelative to the three-dimensional construction of the soft-tissue withthe electromagnetic position of the catheter.

BRIEF DESCRIPTION OF THE DRAWINGS

Various aspects and embodiments of the disclosure are describedhereinbelow with references to the drawings, wherein:

FIG. 1 is a perspective view of one illustrative embodiment of anelectromagnetic navigation (EMN) system incorporating a fluoroscopicimaging device in accordance with the disclosure;

FIG. 2 is a flow chart of a method for detecting a catheter influoroscopic data and eliminating false-positive detections;

FIG. 3A is an illustration of an example of a frame of fluoroscopic datacaptured by a fluoroscopic imaging device showing an extended workingchannel of a catheter assembly positioned within a target region of apatient in accordance with the instant disclosure;

FIG. 3B is an illustration of a resulting image after an initialcatheter detection operator is performed on the frame of FIG. 3A;

FIG. 4A is an illustration of an example of a frame of a fluoroscopicdata captured by a fluoroscopic imaging device showing an extendedworking channel of a catheter assembly positioned within a target regionof a patient in accordance with the instant disclosure;

FIG. 4B is an illustration of a resulting image after secondary catheterdetection operator is performed on the frame of FIG. 4A;

FIG. 5 is an illustration of resulting image frames of fluoroscopicdata, after undergoing initial and secondary catheter detectionoperators, and respective rays of each resulting frame intersecting at acommon intersecting point;

FIG. 6A is an illustration of a resulting image after an initial 3Dreconstruction of a 3D position of a catheter tip;

FIG. 6B is an illustration of a resulting image after a 2D detections ofthe catheter tip of FIG. 6A are reweighted; and

FIG. 7 is a flow chart of a method for updating the position of anavigated catheter relative to a target on a 3D rendering of a patient'sluminal network.

DETAILED DESCRIPTION

In order to navigate tools to a remote soft-tissue target for biopsy ortreatment, both the tool and the target should be visible in some sortof a three dimensional guidance system. The majority of these systemsuse some X-ray device to see through the body. For example, a CT machinecan be used with iterative scans during procedure to provide guidancethrough the body until the tools reach the target. This is a tediousprocedure as it requires several full CT scans, a dedicated CT room andblind navigation between scans. In addition, each scan requires thestaff to leave the room. Another option is a Cone-beam CT machine whichis available in some operation rooms and is somewhat easier to operate,but is expensive and like the CT only provides blind navigation betweenscans, requires multiple iterations for navigation and requires thestaff to leave the room.

Accordingly, there is a need for a system that can achieve the benefitsof the CT and Cone-beam CT three dimensional image guidance without theunderlying costs, preparation requirements, and radiation side effectsassociated with these systems.

The disclosure is directed to a system and method for catheter detectionin fluoroscopic data and constructing local three dimensional volumetricdata, in which small soft-tissue objects are visible, from thefluoroscopic data captured by a standard fluoroscopic imaging deviceavailable in most procedure rooms. The catheter detection andconstructed fluoroscopic-based local three dimensional volumetric datamay be used for guidance, navigation planning, improved navigationaccuracy, navigation confirmation, and treatment confirmation.

FIG. 1 depicts an Electromagnetic Navigation (EMN) system 100 configuredfor reviewing CT image data to identify one or more targets, planning apathway to an identified target (planning phase), navigating an extendedworking channel (EWC) 12 of a catheter guide assembly 40 to a target(navigation phase) via a user interface, and confirming placement of theEWC 12 relative to the target. One such EMN system is theELECTROMAGNETIC NAVIGATION BRONCHOSCOPY® system currently sold byMedtronic PLC. The target may be tissue of interest identified by reviewof the CT image data during the planning phase. Following navigation, amedical instrument, such as a biopsy tool or other tool, may be insertedinto the EWC 12 to obtain a tissue sample (or perform any treatment)from the tissue located at, or proximate to, the target.

As shown in FIG. 1, EWC 12 is part of a catheter guide assembly 40. Inpractice, the EWC 12 is inserted into bronchoscope 30 for access to aluminal network of the patient “P.” Specifically, EWC 12 of catheterguide assembly 40 may be inserted into a working channel of bronchoscope30 for navigation through a patient's luminal network. A distal portionof the EWC 12 includes a sensor 44. The position and orientation of thesensor 44 relative to the reference coordinate system, and thus thedistal portion of the EWC 12, within an electromagnetic field can bederived. Catheter guide assemblies 40 are currently marketed and sold byMedtronic PLC under the brand names SUPERDIMENSION® Procedure Kits, orEDGE™ Procedure Kits, and are contemplated as useable with thedisclosure. For a more detailed description of the catheter guideassemblies 40, reference is made to commonly-owned U.S. Pat. No.9,247,992, filed on Mar. 15, 2013, by Ladtkow et al, U.S. Pat. Nos.7,233,820, and 9,044,254, the entire contents of each of which arehereby incorporated by reference. System 100 and its components aredescribed in greater detail below.

The following description of FIGS. 2-7 provides an exemplary workflow ofusing the components of system 100, including the computing device 125and the fluoroscopic imaging device 110, to accurately detect a catheterin the fluoroscopic data (for example, by eliminating any false-positivedetections). Additionally, the following description of FIGS. 2-7provides an exemplary workflow of using system 100 to construct local,fluoroscopic-based, three dimensional volumetric data of a desiredregion of interest, register the position of the detected catheterrelative to a target with an electromagnetic coordinate system trackingthe catheter, and update a displayed position of the catheter relativeto the target using the registration. The systems and methods describedherein may be useful for visualizing a particular target region of apatient utilizing imaging devices which are commonly located within asurgical setting during EMN procedures, thereby obviating the need forsubsequent MRI or CT scans, and confirming the placement of the catheterrelative to the target by providing an updated and more accurate displayof the catheter in a 3D rendering of a patient's luminal network.

FIG. 2 illustrates a method for detecting a catheter in fluoroscopicdata and eliminating false-positive detections in conjunction with asystem such as the system 100 described in FIG. 1 which will now bedescribed with particular detail. Although the methods illustrated anddescribed herein are illustrated and described as being in a particularorder and requiring particular steps, any of the methods may includesome or all of the steps and may be implemented in any order.Additionally, any or all of the steps of any of the methods describedherein may be carried out by computing device 125 of system 100 or anyother component or combination of components of system 100.

Method 200 begins with step 201, where a catheter (or any surgical tool)is navigated to a region of interest within a patient's luminal network.The navigation in step 201 may utilize the electromagnetic tracking andpathway plans described above. In step 203, a fluoroscopic sweep of thetarget area is performed to acquire fluoroscopic data of the target areawith the catheter positioned therein. In particular, in step 203, afluoroscope is positioned about the patient such that fluoroscopicimages of the target area may be captured along the entire sweep (e.g.ranging from the angles of −30 degrees and +30 degrees).

After the fluoroscopic data is acquired, in step 205 an initial catheterdetection is performed. In the initial catheter detection, computingdevice performs an operator in each 2D frame of the fluoroscopic data todetect possible catheter tip candidates in each 2D frame. In one aspect,step 205 includes utilizing a shallow neural network operator (e.g.,four layers). Utilizing the shallow neural network operator provides theadvantage of fast completion (approximately 1 second for every 32 2Dframes of the fluoroscopic data), but simultaneously provides thedisadvantage of identifying many false-positive catheter tip detections.In particular, FIG. 3A illustrates an example 2D frame 301 offluoroscopic data before a shallow neural network operator is applied instep 205, and FIG. 3B illustrates the resulting image 305 after theshallow neural network operator is applied in step 205 showing all ofthe catheter tip candidates detected. As seen in FIG. 3B, while the truecatheter tip 307 in the frame has been detected as a candidate, manyfalse-positive 309 candidate detections have been made as well.

In step 207, a secondary catheter detection operation is performed todetect catheter tip candidates in each 2D frame of the fluoroscopicdata. In one aspect, step 207 includes utilizing a second neural networkoperator, in this case a deep neural network operator (e.g., elevenlayers). Utilizing the deep neural network operator takes a longerperiod of time to complete when compared to the shallow neural networkoperator, but results in fewer false-positive candidate detections.Additionally, the deep neural network operator alone, though notproviding as many false-positive candidate detections as the shallowneural network operator, in some instances fails to detect the actualcatheter tip as a candidate. FIG. 4A illustrates an example 2D frame 401of fluoroscopic data before a deep neural network operator is applied instep 207, and FIG. 4B illustrates the resulting image 405 after the deepneural network operator is applied in step 207. As seen in FIG. 4B, thetrue catheter tip 407 in the image has been detected as a candidate andonly few false-positive 409 candidate detections have been made.

In view of the above, one benefit of the disclosed method is thatutilizing only one of the initial catheter detection in step 205 or thesecondary catheter detection in step 207 could be insufficient foraccurately identifying the catheter tip in the fluoroscopic data. Asexplained above, if the shallow neural network operator of step 205 wereperformed alone, the catheter will be detected in almost all of the 2Dframes, but many false-positives will be detected as well, leading to aninaccurate catheter detection. Additionally, if the deep neural networkoperator were performed alone, then few false positives will bedetected, but some actual catheter detections will be missed, alsoleading to an inaccurate catheter detection. Accordingly, method 200includes applying both the shallow neural network operator and the deepneural network operator to ensure an accurate catheter detection.Additionally, if the deep neural network has a misdetection in a fluorovideo frame, the catheter won't be detected in that frame (even if itwas found in the initial catheter detection). Thus, one thing thatprevents misdetection of the catheter in this case are valid detectionsfrom other frames, as described in further detail below. Another benefitof utilizing two separate neural networks is that splitting the detectorto an initial network and a second deeper network is may enhance runtimeoptimization as described above.

In one aspect, in step 207, the deep neural network operator considersthe candidates from the initial catheter detection of step 205. That is,in utilizing the candidates detected in the initial catheter detection,it can be ensured that the actual catheter will not be missed in thedeep neural network operator and will always be identified as a cathetercandidate after performance of the secondary catheter detection. Inparticular, the initial catheter detection outputs a “catheterprobability” for each pixel in each fluoroscopic frame. Only pixels withhigh probability (above some fixed threshold) are considered as cathetercandidates after the initial catheter detection. An image patch isextracted around each pixel candidate (having the catheter probabilityexceeding the threshold) and is input to the second catheter detection(e.g., the deep neural network).

In step 209, false-positive detections are eliminated by reconstructing3D position of the catheter tip. In particular, with reference to FIG.5, the intersecting point 505 of the rays 503 a, 503 b, 503 c . . . 503n extending from the candidates of each 2D frame 501 a, 501 b, 501 c . .. 501 n is the 3D reconstructed position of the catheter tip. In step209, when the number of rays intersecting at the same point for aparticular catheter candidate exceeds a preconfigured threshold, it isdetermined that the candidate is not a false-positive and is actuallythe catheter tip. In particular, in order to confirm that the candidateis not a false-positive detection, a certain numbers of rays must beintersecting at the same intersection point. When the number of raysintersecting at the same point for a particular catheter candidate isbelow a preconfigured threshold, then it is determined that thecandidate is a false-positive detection and not the catheter tip. Method200 may optionally also include step 211, where the process ofeliminating false-positive candidate detections is iteratively repeated.Iteratively repeating the elimination of false-positive candidatedetections in step 211, enables the preconfigured threshold forintersecting rays to be set at an optimal value, leading to morereliable and accurate elimination of false-positive candidatedetections.

In step 213, the catheter candidate detections are reweighed accordingto their distance from the projected 3D point in step 209. Inparticular, with reference to FIGS. 6A and 6B, FIG. 6A illustrates anexample image 601 after steps 205-209 are completed and FIG. 6Billustrates an image 605 after the reweighing of step 213 in performed.The image 601 in FIG. 6A includes as catheter tip candidates, the actualcatheter tip 607 and a false-positive 603. The image 605 in FIG. 6Bincludes only the actual catheter tip 607 as a candidate and does notinclude any false-positives. In step 213, each voxel is assigned aweight according to the number of rays intersecting through the voxeland the distance of the voxel from the 3D position of the catheter tipidentified in step 209. In step 213, any candidate that is far indistance from the 3D position of the catheter tip identified in step 209will be assigned a lower weight and will be identified as afalse-positive candidate detection and will be removed. On the otherhand, any candidate that is close in distance to the 3D position of thecatheter tip identified in step 209 will be assigned a higher weight andwill be identified as a catheter tip candidate detection and willremain.

FIG. 7 illustrates a method for performing an electromagnetic surgicalnavigation procedure in conjunction with a system such as the system 100described in FIG. 1 utilizing method 200, which will now be describedwith particular detail and will be referred to as method 700. Althoughthe methods illustrated and described herein are illustrated anddescribed as being in a particular order and requiring particular steps,any of the methods may include some or all of the steps and may beimplemented in any order. Additionally, any or all of the steps of anyof the methods described herein may be carried out by computing device125 of system 100 or any other component or combination of components ofsystem 100.

Method 700 begins at step 701 where the electromagnetic position of acatheter is tracked during navigation of the catheter to a target areawithin a patient's luminal network. In step 703, the trackedelectromagnetic position of the catheter is displayed during thenavigation on a display of a CT-based 3D rendering of the patient'sluminal network. Displaying the electromagnetic position of the catheteron the 3D rendering of the patient's luminal network assists theclinician in navigating to the region of interest or target area. Step703 may additionally include displaying a predetermined route or path tofollow through the patient's luminal network to navigate to the targetarea. Such a display of the pathway or route enables a clinician toidentify when the catheter has deviated from the desired path, theprogress of the navigation, and when the target is reached.

Once the catheter is navigated to the target area as displayed in step703, it is possible that the actual position of the catheter within thepatient's luminal network is not exactly as displayed in step 703,because the displayed position of the catheter is based only on theelectromagnetically tracked position of the catheter's sensor and isdisplayed over an old CT-data set. Targets within the target area, andother such anatomical features, may have changed since the time ofacquisition of the CT data and the electromagnetic tracking is subjectto interference and unreliability. Additionally, the display in step 703merely shows a representation of the catheter, and not the actualcatheter within the luminal network relative to structures therein. Tomore accurately display the actual position of the catheter within thepatient's actual luminal network, method 700 proceeds to step 705.

In step 705, a fluoroscopic sweep of the target area is performed toacquire fluoroscopic data of the target area with the catheterpositioned therein. In particular, in step 705, a fluoroscope ispositioned about the patient such that fluoroscopic images of the targetarea may be captured along a sweep (e.g. ranging from the angles of −30degrees and +30 degrees). In step 707, the catheter tip is detectedthroughout the frames of the fluoroscopic data. In one aspect, step 707utilizes some or all of the steps of method 200 for the catheterdetection.

In step 709, a fluoroscopic-based 3D rendering of the target area withthe catheter positioned therein is constructed. Such afluoroscopic-based 3D construction enables visualization of objects thatare not otherwise visible in the fluoroscopic data itself. For example,small soft-tissue objects, such as tumors or lesions, are not visible inthe fluoroscopic data, but are visible in the fluoroscopic-based 3Drendering of the target area. Further details regarding the constructionof step 709 may be found in U.S. Patent Application Publication No.2017/0035380, filed Aug. 1, 2016, entitled System and Method forNavigating to Target and Performing Procedure on Target UtilizingFluoroscopic-Based Local Three Dimensional Volume Reconstruction, andU.S. Patent Application Publication No. 2017/0035379, filed Aug. 1,2016, entitled System and Method for Local Three Dimensional VolumeReconstruction Using a Standard Fluoroscope, the entire contents of eachof which are incorporated by reference herein.

In step 711, position data of the catheter relative to the target in thetarget area in each frame of the fluoroscopic data is acquired. Inparticular, the construction of the fluoroscopic-based 3D rendering instep 709 enables the visibility of the target (e.g., soft tissue object,lesion, tumor, etc.) and the location of the target to be determined orotherwise ascertained. As described above, prior to the construction ofthe fluoroscopic-based 3D rendering in step 709, the soft-tissue objects(e.g., target, tumor, lesion, etc.) are not visible and the location ofthe object(s), relative to the catheter cannot be determined. In oneaspect, in step 711, the position, orientation, and distance of thecatheter relative to the target are determined for each slice of thefluoroscopic-based 3D rendering. Additionally, the position dataacquired in step 711 may be correlated to the fluoroscopic data acquiredin step 705.

In step 713, the position data acquired in step 711 is registered to theelectromagnetic tracked coordinates of the catheter. In one aspect, instep 713, the system displays the catheter on the CT data by employingan “Antenna-to-CT” registration, where the catheter position (andoptionally orientation) in antenna coordinates is continuouslycalculated by the electromagnetic localization algorithms. The positionis then transformed to CT coordinates. This registration may be based onthe first bifurcations of the airway tree (main carina and bronchi). Dueto this (and other factors, such as patient sedation, and differentpatient pose during CT and bronchoscopic procedure) the registration isless accurate in the periphery of the lungs. The registration iscorrected for the target area by composing one or more of the followingregistrations: 1) Antenna-to-fluoro, where the catheter position isknown in both antenna coordinates (using electromagnetic localization)and fluoro coordinates (automatic catheter detection described in thepreceding steps), and is used as a basis for the registration; and/or 2)Fluoro-to-CT, where the target's position is marked on the CT datapre-operatively and then also marked on the fluoro-based 3Dreconstruction inter-operatively. After performing calculations for theabove registrations, a new catheter position in antenna coordinates istransformed to fluoro coordinates by employing the “Antenna-to-fluoro”registration, which is then transformed to CT coordinates by employingthe “Fluoro-to-CT” registration.

In step 715, the displayed position of the catheter on the CT-based 3Drendering of the patient's luminal network is updated using the positiondata acquired in step 711. In particular, the position, orientation, anddistance of the catheter relative to the target in thefluoroscopic-based 3D data set are compared to the position,orientation, and distance of the displayed catheter relative to thetarget in the CT-based 3D rendering. In one aspect, the orientation,position, and distance of the displayed catheter relative to the targetin the CT-based rendering is updated to correspond or match with theorientation, position, and distance of the catheter relative to thetarget in the fluoroscopic-based 3D data set.

Referring back to FIG. 1, EMN system 100 generally includes an operatingtable 20 configured to support a patient “P;” a bronchoscope 30configured for insertion through the patient's “P's” mouth into thepatient's “P's” airways; monitoring equipment 120 coupled tobronchoscope 30 (e.g., a video display, for displaying the video imagesreceived from the video imaging system of bronchoscope 30); a trackingsystem 50 including a tracking module 52, a plurality of referencesensors 54 and a transmitter mat 56; and a computing device 125including software and/or hardware used to facilitate identification ofa target, pathway planning to the target, navigation of a medicalinstrument to the target, and confirmation of placement of an EWC 12, ora suitable device therethrough, relative to the target.

A fluoroscopic imaging device 110 capable of acquiring fluoroscopic orx-ray images or video of the patient “P” is also included in thisparticular aspect of system 100. The fluoroscopic data (e.g., images,series of images, or video) captured by the fluoroscopic imaging device110 may be stored within the fluoroscopic imaging device 110 ortransmitted to computing device 125 for storage, processing, anddisplay. Additionally, the fluoroscopic imaging device 110 may moverelative to the patient “P” so that images may be acquired fromdifferent angles or perspectives relative to the patient “P” to create afluoroscopic video from a fluoroscopic sweep. Fluoroscopic imagingdevice 110 may include a single imaging device or more than one imagingdevice. In embodiments including multiple imaging devices, each imagingdevice may be a different type of imaging device or the same type.Further details regarding the imaging device 110 are described in U.S.Pat. No. 8,565,858, which is incorporated by reference in its entiretyherein.

Computing device 125 may be any suitable computing device including aprocessor and storage medium, wherein the processor is capable ofexecuting instructions stored on the storage medium. The computingdevice 125 may further include a database configured to store patientdata, CT data sets including CT images, fluoroscopic data sets includingfluoroscopic images and video, navigation plans, and any other suchdata. Although not explicitly illustrated, the computing device 125 mayinclude inputs, or may otherwise be configured to receive, CT data sets,fluoroscopic images/video and other data described herein. Additionally,computing device 125 includes a display configured to display graphicaluser interfaces.

With respect to the planning phase, computing device 125 utilizespreviously acquired CT image data for generating and viewing a threedimensional model of the patient's “P's” airways, enables theidentification of a target on the three dimensional model(automatically, semi-automatically, or manually), and allows fordetermining a pathway through the patient's “P's” airways to tissuelocated at and around the target. More specifically, CT images acquiredfrom previous CT scans are processed and assembled into a threedimensional CT volume, which is then utilized to generate a threedimensional model of the patient's “P's” airways. The three dimensionalmodel may be displayed on a display associated with computing device125, or in any other suitable fashion. Using computing device 125,various views of the three dimensional model or enhanced two dimensionalimages generated from the three dimensional model are presented. Theenhanced two dimensional images may possess some three dimensionalcapabilities because they are generated from three dimensional data. Thethree dimensional model may be manipulated to facilitate identificationof target on the three dimensional model or two dimensional images, andselection of a suitable pathway (e.g., route to be following duringnavigation) through the patient's “P's” airways to access tissue locatedat the target can be made. Once selected, the pathway plan, threedimensional model, and images derived therefrom, can be saved andexported to a navigation system for use during the navigation phase(s).One such planning software is the ILOGIC® planning suite currently soldby Medtronic PLC.

With respect to the navigation phase, a six degrees-of-freedomelectromagnetic tracking system 50, e.g., similar to those disclosed inU.S. Pat. Nos. 8,467,589, 6,188,355, and published PCT Application Nos.WO 00/10456 and WO 01/67035, the entire contents of each of which areincorporated herein by reference, or other suitable positioningmeasuring system, is utilized for performing registration of the imagesand the pathway for navigation, although other configurations are alsocontemplated. Tracking system 50 includes a tracking module 52, aplurality of reference sensors 54, and a transmitter mat 56. Trackingsystem 50 is configured for use with a sensor 44 of catheter guideassembly 40 to track the electromagnetic position thereof within anelectromagnetic coordinate system.

Transmitter mat 56 is positioned beneath patient “P.” Transmitter mat 56generates an electromagnetic field around at least a portion of thepatient “P” within which the position of a plurality of referencesensors 54 and the sensor element 44 can be determined with use of atracking module 52. One or more of reference sensors 54 are attached tothe chest of the patient “P.” The six degrees of freedom coordinates ofreference sensors 54 are sent to computing device 125 (which includesthe appropriate software) where they are used to calculate a patientcoordinate frame of reference. Registration, as detailed below, isgenerally performed to coordinate locations of the three dimensionalmodel and two dimensional images from the planning phase with thepatient's “P's” airways as observed through the bronchoscope 30, andallow for the navigation phase to be undertaken with precise knowledgeof the location of the sensor 44, even in portions of the airway wherethe bronchoscope 30 cannot reach. Further details of such a registrationtechnique and their implementation in luminal navigation can be found inU.S. Patent Application Publication No. 2011/0085720, the entire contentof which is incorporated herein by reference, although other suitabletechniques are also contemplated.

Registration of the patient's “P's” location on the transmitter mat 56is performed by moving sensor 44 through the airways of the patient's“P.” More specifically, data pertaining to locations of sensor 44, whileEWC 12 is moving through the airways, is recorded using transmitter mat56, reference sensors 54, and tracking module 52. A shape resulting fromthis location data is compared to an interior geometry of passages ofthe three dimensional model generated in the planning phase, and alocation correlation between the shape and the three dimensional modelbased on the comparison is determined, e.g., utilizing the software oncomputing device 125. In addition, the software identifies non-tissuespace (e.g., air filled cavities) in the three dimensional model. Thesoftware aligns, or registers, an image representing a location ofsensor 44 with a the three dimensional model and two dimensional imagesgenerated from the three dimension model, which are based on therecorded location data and an assumption that sensor 44 remains locatedin non-tissue space in the patient's “P's” airways. Alternatively, amanual registration technique may be employed by navigating thebronchoscope 30 with the sensor 44 to pre-specified locations in thelungs of the patient “P”, and manually correlating the images from thebronchoscope to the model data of the three dimensional model.

Following registration of the patient “P” to the image data and pathwayplan, a user interface is displayed in the navigation software whichsets forth the pathway that the clinician is to follow to reach thetarget. One such navigation software is the ILOGIC® navigation suitecurrently sold by Medtronic PLC.

Once EWC 12 has been successfully navigated proximate the target asdepicted on the user interface, the EWC 12 is in place as a guidechannel for guiding medical instruments including without limitation,optical systems, ultrasound probes, marker placement tools, biopsytools, ablation tools (i.e., microwave ablation devices), laser probes,cryogenic probes, sensor probes, and aspirating needles to the target.

With the above-described updated position of the catheter relative to atarget in a target area, a clinician is able to more accurately navigateto a region of interest or target within a patient's luminal network.Errors in the electromagnetically tracked and displayed location of thecatheter in the CT-based 3D rendering of the patient's luminal networkare corrected based on near real-time position data extracted fromfluoroscopic-based 3D data of target area.

From the foregoing and with reference to the various figure drawings,those skilled in the art will appreciate that certain modifications canalso be made to the disclosure without departing from the scope of thesame. For example, although the systems and methods are described asusable with an EMN system for navigation through a luminal network suchas the lungs, the systems and methods described herein may be utilizedwith systems that utilize other navigation and treatment devices such aspercutaneous devices. Additionally, although the above-described systemand method is described as used within a patient's luminal network, itis appreciated that the above-described systems and methods may beutilized in other target regions such as the liver. Further, theabove-described systems and methods are also usable for transthoracicneedle aspiration procedures.

Detailed embodiments of the disclosure are disclosed herein. However,the disclosed embodiments are merely examples of the disclosure, whichmay be embodied in various forms and aspects. Therefore, specificstructural and functional details disclosed herein are not to beinterpreted as limiting, but merely as a basis for the claims and as arepresentative basis for teaching one skilled in the art to variouslyemploy the disclosure in virtually any appropriately detailed structure.

As can be appreciated a medical instrument such as a biopsy tool or anenergy device, such as a microwave ablation catheter, that ispositionable through one or more branched luminal networks of a patientto treat tissue may prove useful in the surgical arena and thedisclosure is directed to systems and methods that are usable with suchinstruments and tools. Access to luminal networks may be percutaneous orthrough natural orifice using navigation techniques. Additionally,navigation through a luminal network may be accomplished usingimage-guidance. These image-guidance systems may be separate orintegrated with the energy device or a separate access tool and mayinclude MRI, CT, fluoroscopy, ultrasound, electrical impedancetomography, optical, and/or device tracking systems. Methodologies forlocating the access tool include EM, IR, echolocation, optical, andothers. Tracking systems may be integrated to an imaging device, wheretracking is done in virtual space or fused with preoperative or liveimages. In some cases the treatment target may be directly accessed fromwithin the lumen, such as for the treatment of the endobronchial wallfor COPD, Asthma, lung cancer, etc. In other cases, the energy deviceand/or an additional access tool may be required to pierce the lumen andextend into other tissues to reach the target, such as for the treatmentof disease within the parenchyma. Final localization and confirmation ofenergy device or tool placement may be performed with imaging and/ornavigational guidance using a standard fluoroscopic imaging deviceincorporated with methods and systems described above.

While several embodiments of the disclosure have been shown in thedrawings, it is not intended that the disclosure be limited thereto, asit is intended that the disclosure be as broad in scope as the art willallow and that the specification be read likewise. Therefore, the abovedescription should not be construed as limiting, but merely asexemplifications of particular embodiments. Those skilled in the artwill envision other modifications within the scope and spirit of theclaims appended hereto.

1. (canceled)
 2. A system for performing an electromagnetic surgicalnavigation procedure, comprising: a catheter including a sensor on adistal portion of the catheter; a tracking system configured todetermine the position and orientation of the distal portion of thecatheter; and a computing device operably coupled to the tracking systemand the catheter, the computing device configured to: acquirefluoroscopic data from a fluoroscopic sweep of a target area, thefluoroscopic data including 2D fluoroscopic frames of the target areacaptured from different perspectives; perform an initial catheterdetection for catheter tip candidates in a plurality of 2D frames of thefluoroscopic data; perform a secondary catheter detection for cathetertip candidates in the plurality of 2D frames of the fluoroscopic data;determine an intersection point of rays extending from the detectedcatheter tip candidates in each of the plurality of 2D frames of thesecondary catheter detection; and eliminate as false-positives thecatheter tip candidates of the secondary catheter detections whose raysdo not pass through the intersecting point.
 3. The system according toclaim 2, wherein the computing device configured to perform an initialcatheter detection for catheter tip candidates in the plurality of 2Dframes of the fluoroscopic data include applying a shallow neuralnetwork operator.
 4. The system according to claim 2, wherein thecomputing device configured to perform a secondary catheter detectionfor catheter tip candidates in the plurality of 2D frames for thefluoroscope data includes applying a deep neural network operator. 5.The system according to claim 2, wherein the computing device is furtherconfigured to iteratively determine an intersecting point of raysextending from the detected catheter tip candidates in each of theplurality of 2D frames of the second catheter detection and eliminatefalse-positive catheter tip candidates of the second catheter tipdetections.
 6. The system according to claim 2, wherein the computingdevice is further configured to display a user interface for manuallyselecting the catheter tup in a 2D fluoroscopic frame of thefluoroscopic data prior to performing an initial catheter detection forcatheter tip candidates in each 2D frame of the fluoroscopic data. 7.The system according to claim 2, wherein the computing device is furtherconfigured to reweigh the catheter tip candidates of the secondarycatheter detection based on a distance of the catheter tip candidatefrom a projected 3D point.
 8. The system according to claim 7, whereinthe computing device configured to reweigh the catheter tip candidatesincludes decreasing a weight of a pixel corresponding to a candidatewhen the distance of the catheter tip candidate is far from theprojected 3D point.
 9. A system for performing an electromagneticsurgical navigation procedure, comprising: a catheter including a sensoron a distal portion of the catheter; a tracking system configured todetermine the position and orientation of the distal portion of thecatheter; a fluoroscope operably coupled to the tracking system; and acomputing device operably coupled to the tracking system, fluoroscope,and the catheter, the computing device configured to: acquirefluoroscopic data from the fluoroscope, the fluoroscopic data including2D fluoroscopic frames of a target area; perform an initial catheterdetection for catheter tip candidates in a plurality of 2D frames of thefluoroscopic data; perform a secondary catheter detection for cathetertip candidates in the plurality of 2D frames of the fluoroscopic data;determine an intersection point of rays extending from the detectedcatheter tip candidates in each of the plurality of 2D frames of thesecondary catheter detection; and eliminate as false-positives thecatheter tip candidates of the secondary catheter detections whose raysdo not pass through the intersecting point.
 10. The system according toclaim 9, wherein the computing device configured to perform an initialcatheter detection for catheter tip candidates in the plurality of 2Dframes of the fluoroscopic data include applying a shallow neuralnetwork operator.
 11. The system according to claim 9, wherein thecomputing device configured to perform a secondary catheter detectionfor catheter tip candidates in the plurality of 2D frames for thefluoroscope data includes applying a deep neural network operator. 12.The system according to claim 9, wherein the computing device is furtherconfigured to iteratively determine an intersecting point of raysextending from the detected catheter tip candidates in each of theplurality of 2D frames of the second catheter detection and eliminatefalse-positive catheter tip candidates of the second catheter tipdetections.
 13. The system according to claim 9, wherein the computingdevice is further configured to display a user interface for manuallyselecting the catheter tup in a 2D fluoroscopic frame of thefluoroscopic data prior to performing an initial catheter detection forcatheter tip candidates in each 2D frame of the fluoroscopic data. 14.The system according to claim 9, wherein the computing device is furtherconfigured to reweigh the catheter tip candidates of the secondarycatheter detection based on a distance of the catheter tip candidatefrom a projected 3D point.
 15. The system according to claim 14, whereinthe computing device configured to reweigh the catheter tip candidatesincludes decreasing a weight of a pixel corresponding to a candidatewhen the distance of the catheter tip candidate is far from theprojected 3D point.
 16. A system for performing an electromagneticsurgical navigation procedure, comprising: a catheter including a sensoron a distal portion of the catheter; a tracking system configured todetermine the position and orientation of the distal portion of thecatheter; and a computing device operably coupled to the trackingsystem, fluoroscope, and the catheter, the computing device configuredto: acquire fluoroscopic data from a fluoroscopic sweep of the targetarea, the fluoroscopic data including 2D fluoroscopic frames of thetarget area captured from the different perspectives; perform an initialcatheter detection for catheter tip candidates in a plurality of the 2Dframes of the fluoroscopic data; perform a secondary catheter detectionfor catheter tip candidates in the plurality of the 2D frames of thefluoroscopic data; determine an intersecting point of rays extendingfrom the detected catheter tip candidates in each of the plurality of 2Dframes of the secondary catheter detection; eliminate as false positivedetections the catheter tip candidates of the secondary catheterdetections whose rays do not pass through the intersecting point;construct a fluoroscopic-based three dimensional volumetric data of thetarget area from the acquired fluoroscopic data, the fluoroscopic-basedthree dimensional volumetric data including a three-dimensionalconstruction of a soft-tissue target in the target area; identify foreach 2D frame of the fluoroscopic data the intersecting point as a 3Dposition of the catheter tip and the intersecting points relativeposition to the three-dimensional construction of the soft-tissuetarget; and display a position of the catheter tip and thethree-dimensional construction of the soft-tissue.
 17. The systemaccording to claim 16, wherein the computing device configured toperform an initial catheter detection for catheter tip candidates in theplurality of 2D frames of the fluoroscopic data include applying ashallow neural network operator.
 18. The system according to claim 16,wherein the computing device configured to perform a secondary catheterdetection for catheter tip candidates in the plurality of 2D frames forthe fluoroscope data includes applying a deep neural network operator.19. The system according to claim 16, wherein the computing device isfurther configured to iteratively determine an intersecting point ofrays extending from the detected catheter tip candidates in each of theplurality of 2D frames of the second catheter detection and eliminatefalse-positive catheter tip candidates of the second catheter tipdetections.
 20. The system according to claim 16, wherein the computingdevice is further configured to display a user interface for manuallyselecting the catheter tup in a 2D fluoroscopic frame of thefluoroscopic data prior to performing an initial catheter detection forcatheter tip candidates in each 2D frame of the fluoroscopic data. 21.The system according to claim 16, wherein the computing device isfurther configured to reweigh the catheter tip candidates of thesecondary catheter detection based on a distance of the catheter tipcandidate from a projected 3D point.